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Executive summary 


The project 


The MathsFlip intervention aimed to improve the attainment of pupils in Years 5 and 6. The programme, 
developed by Shireland Collegiate Academy, used a ‘flipped learning’ approach involving pupils learning 
core content online, outside of class time, and then participating in activities in class to reinforce their 
learning. The programme used an online learning environment which provided teachers and pupils with 
resources for learning mathematics outside the classroom, allowed collaborative communication 
between colleagues and pupils, and provided information to teachers on pupils’ progress prior to 
planning and teaching a lesson. Staff from Shireland trained Year 5 teachers from 12 primary schools 
in the West Midlands over two days and provided some ongoing support. 


The project was a randomised controlled trial involving 24 schools: 12 schools received the intervention 
from April 2014 until April 2015 using MathsFlip to deliver lessons at the start of a new maths topic for 
the cohort of pupils in Year 5; 12 schools acted as control schools delivering maths lessons in the usual 
way. Impact was measured using pupils’ Key Stage 2 (KS2) maths scores in summer 2015. The process 
evaluation involved lesson observations and collecting teachers’ and pupils’ perceptions of the 
programme through interviews with staff, focus groups, and online teacher surveys. The project was 
funded as part of the EEF Digital Technology funding round in collaboration with the Nominet Trust. 


Key conclusions 


1. Children in MathsFlip schools made the equivalent of one additional months’ progress in maths, 
on average, compared to children in comparison schools. The three padlock security rating 
means this result is moderately secure. 


The impact on maths was slightly higher for children eligible for free school meals (‘FSM pupils’) 
than for all children in the trial. These results are less secure than the overall findings because 
of the smaller number of pupils. 


Children in MathsFlip schools made three additional months’ progress in reading and writing, on 
average, compared to children in the other schools. However, this result should be treated with 
caution. First, there is not an obvious route by which this maths intervention could improve 
literacy results so much more than maths results. Second, the relatively small number of schools 
involved increases the likelihood that we would see a result like this just by chance rather than 
due to the intervention itself. 


The majority of teachers in the trial were very positive about the flipped learning approach and 
the technical and professional support they received from Shireland staff. The process evaluation 
suggests this support was necessary for successful implementation. 


Some teachers experienced technical problems with the online platform; these were generally 
dealt with quickly by the developers. Some pupils did not have internet access at home: this led 
some schools to set up homework clubs providing online access. 


EEF security rating 


These findings have moderate security. This trial was an efficacy trial which tested whether the 
intervention worked under developer-led conditions in a number of schools. The trial was a well- 
designed two-armed randomised controlled trial. Relatively few pupils—only 7% of those who started 
the trial—were not included the final analysis. The pupils in MathsFlip schools were similar to those in 
the comparison schools in terms of levels of FSM eligibility and prior attainment. However, the trial was 


3 
Education Endowment Foundation 


Flipped Learning 


powered to detect an effect of 0.37, higher than the expected EEF maximum of 0.2. Therefore, two 
padlocks were lost, reducing the security of the trial. 


Additional findings 


MathsFlip appeared to have only a small impact on KS2 mathematics achievement at the end of Year 
6. The impact observed for KS2 reading and writing outcomes was higher, however this result should 
be treated with caution for two reasons. First, there is not an obvious route by which this maths 
intervention could improve literacy results so much more than maths results. Second, the relatively small 
number of schools involved increases the likelihood that we would see a result like this just by chance 
rather than due to the intervention itself. 


Despite some initial start-up problems and challenges with teachers’ lack of familiarity with the 
technology, both teachers and pupils reported very positive perceptions of the intervention. Some 
teachers did comment that it demanded a high level of independence from pupils and support from 
parents to make the approach work. There were issues with accessing the online platform at home for 
some pupils. Teachers positively rated the support that they received from the Shireland implementation 
support staff. 


Cost 


MathsFlip cost about £4,400 per school, or £147 per pupil per year when averaged over three years. 
Schools also need to meet the cost of staff cover for 3 days of training for each participating teacher. 
Participating pupils also need access to the internet to complete activities outside of class time, either 
at school or at home. 


Summary of impact on main outcomes 


Effect size Estimated FEF 
Outcome! 
95% confidence §_ months’ Novel -value _ securit EEF cost 
p : 


interval) progress 


Group pupils EN ale] 


rating 


0.09 
(-0.28 to 0.46) 


KS2 Maths 


KS2 Maths 0.10 
FSM pupils (-0.21 to 0.40) 


Education Endowment Foundation 


Flipped Learning 


Introduction 


Intervention 


‘Flipped learning’ (often referred to as the ‘flipped classroom’ or ‘blended learning’) is an approach in 
which pupils are given assignments to do at home on computers or tablets and teachers collect feedback 
prior to the lesson. These may include video lessons made by the teacher or online activities related to 
the content. This pre-learning experience enables the teacher to use classroom time for activities such 
as co-operative learning, problem-solving, projects, and attending to individual difficulties. Prior to the 
lesson, pupils access concepts and acquire knowledge at home, thus teachers spend more lesson time 
working on problem-solving and on higher order thinking tasks in class to help develop pupils’ thinking. 


The intervention was delivered from April 2014 to April 2015 to the treatment group and from April 2015 
to the control schools (with the new Year 5 pupils). The MathsFlip programme was developed by 
Shireland Collegiate Academy taking as a basis the flipped learning approach that was implemented by 
Shireland teachers. 


The key element of the MathsFlip programme was the online learning environment, based on the 
Microsoft Office 365 platform, which provided teachers and pupils with resources for learning 
mathematics outside the classroom. The learning platform was developed by Shireland staff and 
consists of two main elements: (1) the class site and (2) the learning zone. The class site was a 
personalised page for each class. It contained individual areas where teachers could upload resources 
for colleagues and pupils to access (such as videos, links, and documents) and shared areas that 
allowed collaborative communication among the members of the class (for example, announcements, 
calendar, discussions). 


The learning zone was a resource bank based on the learning objectives for mathematics according to 
the national curriculum. The learning zone allowed teachers to access resources specific to the topics 
they selected. These selected resources were then shared with the pupils through class sites. 
Resources were organised by topic, Year (that is, level of difficulty), and presentation type (for example, 
video, Power Point). Resources included maths games and teaching programmes such as Mangahigh 
and MyMaths. 


The MathsFlip approach depended on the pupils having access to laptop computers (Shirelands 
supplied 30 laptops to each school which pupils could take home). In multiple form entry schools, staff 
had to manage this process to ensure equality of access for pupils. It also depended on pupils having 
internet access at home to work on learning activities after school. Where this was not the case, schools 
provided lunchtime, before-school, or after-school sessions during which pupils could complete these 
activities. 


Year 5 teachers were trained in the use of the MathsFlip platform in two hands-on day-long sessions. 
This was repeated with Year 6 teachers as pupils progressed to the next academic year. The project 
directors, project manager, and the e-learning team (members of the Shireland Collegiate Academy 
staff) delivered the training using the MathsFlip methodology. The aims of the training were to: 


e introduce the flipped learning methodology; 
e explain how MathsFlip could be used to support the delivery of mathematics; and 


e access the MathsFlip online area (the learning zone) and work with the different tools (for 
example, to enable the embedding of videos or uploading work). 


On the first day of the training, teachers were introduced to the flipped learning approach and shown 
how this might look in mathematics. They were also shown how to use the MathsFlip resources in their 
mathematics teaching. Following an evaluation at the end of the day, the second day focused on 
technical support and how to use the class sites. For the second day, teachers self-selected the right 
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level of training based on their previous knowledge and confidence of flipped learning and their use of 
technology. These initial sessions were followed up by a further session six months later where teachers 
evaluated the kind of extra support they needed resulting in the creation of a support plan to meet their 
individual needs. As a result, one large school with four Year 5 teachers asked for additional training; in 
response, the project manager attended the school to deliver two more sessions. 


In addition to the initial training, treatment schools received ongoing support. The project manager made 
regular visits to the treatment schools to help them with the planning and delivery of some activities. 
She visited every school and helped teachers to write an action plan to think about how the programme 
could be implemented in their school. She provided feedback on the lessons and coached teachers who 
needed extra support. Furthermore, hands-on workshops were organised in schools to explain the 
approach to parents and to show them how to access the resources and how to assist their children in 
using the learning materials. 


Background evidence 


The flipped learning approach has been widely used in the last decade, largely in secondary and higher 
education contexts (Bentley, Allan and Belton, 2014; Dunn, 2013; Educause, 2012; Flipped Learning 
Network, 2014; Vincent, 2013). Opinions exist, published mainly in blogs, teachers’ forums, and 
practitioner-oriented magazines, suggesting that the approach might work in other settings (Bergmann 
and Sams, 2012; Fulton, 2012, Moran and Young, 2015). Teachers all around the world are trying the 
approach in their classrooms and many report positive perceptions in the Flipped Learning Network 
(FLN). 


A recent review suggests rationales for why flipped learning might improve learning, but they do not 
present actual comparisons of flipped learning and traditional teaching (Hamdan et al., 2013). Most of 
the studies are from higher education settings. However, caution should be employed as sometimes 
students can become disengaged with flipped or blended learning programmes, with one study 
suggesting that up to 25% of pupils had become disengaged with the approach (Loch and Borland, 
2014). If students do not complete the prerequisite activities or readings before class they are less likely 
to be able to engage with the in-class activity. Although the flipped learning methodology is now used 
in the U.K. it has not been assessed rigorously in a U.K. context. Furthermore, there do not seem to 
exist any rigorous studies with comparison groups at the primary education level. Therefore there are 
no studies to base effect size estimates on. It is important to conduct a study to determine if there is 
evidence behind what is becoming a popular approach. 


Given the popularity of flipped learning and the weakness of the evidence base, the EEF and Nominet 
Trust co-funded a one year randomised controlled trial (RCT) to provide some high quality evidence on 
the impact of flipped learning in English schools. This was part of a funding round focused on digital 
technology. 


Shireland Collegiate Academy is widely acknowledged as a leading school in its use of technology. 
Shireland had been using technology and flipped learning in its classrooms for several years and 
believed that it was able to take the model to other schools. As such, it developed the MathsFlip project 
to train primary school teachers to implement a flipped learning approach to maths with Year 5 and Year 
6 pupils. 


An efficacy trial was conducted to provide evidence of impact using a robust RCT design. 
Evaluation objectives 


The Institute for Effective Education (IEE) conducted an impact and process evaluation to rigorously 
test the effectiveness of the intervention by means of a randomised controlled trial. This efficacy 
evaluation was designed to assess whether flipped learning had an impact on pupils’ numeracy 
attainment and pupils’ performance in mathematics at Key Stage 2. A process evaluation was also 
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conducted to develop an understanding of the MathsFlip approach and assess teacher and pupil 
perceptions of it. 


Ethical review 


The evaluation team obtained ethical approval from the Department of Education, University of York 
Ethical Review Panel on 29 January 2013. Headteachers signed an agreement outlining the main 
commitments of the three parties in the study: the school, the project developers, and the evaluators. 
The evaluation team provided information and opt-out consent forms for parents and guardians. 


Data management was in accordance with the Data Protection Act (1998). The trial database is securely 
held and maintained on the University of York’s research data protection server, with non-identifiable 
data. Confidentiality is maintained and no one outside the trial team has access to the database. Data 
was checked for missing information and double entries. All outputs were anonymized so that no schools 
or students can be identified in any report or dissemination of results. 


Project team 


Dr Peter Rudd, Principal Investigator (retired January, 2016). 
Dr Alaidde Berenice Villanueva Aguilera, Research Associate. 
Ms Louise Elliott, Data Manager. 


Professor Bette Chambers (took over write-up of final report February, 2016). 
Trial registration 


This trial was registered at ISRCTN, number 20851469. 


https:/Awww.isrctn.com/search?q=&filters=intervention%3AMathsFlip%2CfunderName%3AEducation+ 
Endowment+Foundation&searchType=advanced-search 
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Methods 


Trial design 


A two-armed randomised controlled efficacy trial was carried out. Randomisation at school level was 
preferred over randomisation at class level to avoid ‘contamination’ within schools and increase the 
possibilities of collaboration among teachers within each school. 


Schools participating in the project paid £1,000 each and received training and support for their Year 5 
teachers from the MathsFlip team from Shireland Collegiate Academy in spring 2014 for the treatment 
schools, or after the evaluation was completed for the control schools. These teachers implemented the 
MathsFlip approach from April 2014 through to April 2015. The developers intended that Year 5 teachers 
would follow their pupils into Year 6, however this is not common practice in schools so most pupils in 
Year 6 had different teachers (who were also trained in MathsFlip). Shireland responded by training the 
teachers in Year 6 at the end of July and beginning of September. This made the intervention a year- 
long treatment spanning two academic years. However, limited engagement from Year 5 teachers who 
only had one term with pupils did delay the embedding of the approach with these pupils. 


Teachers in the control group delivered ‘business as usual’ maths lessons. Year 5 teachers in control 
schools received the training and support from April 2015 and worked with their current Year 5 pupils 
who then continued with the approach into Year 6. 


Participant selection 


All primary schools with high proportions of pupils with FSM eligibility in Birmingham and the Black 
Country area were eligible to express an interest in the project. Schools were invited to participate by 
Shireland Collegiate Academy project staff. Although the schools were geographically close they were 
not a network or a fixed group prior to participation in the project. 


Fifty-two schools expressed an interest in participating in the study; 24 were selected by the IEE, 
choosing those that had low prior attainment and high levels of socio-economic deprivation (based on 
the proportion of pupils receiving free school meals). 


Prior to randomisation, a launch event took place at the Botanic Gardens in Birmingham. Staff members 
from all 24 schools selected for the project were invited to meet the project partners (funder, deliverer, 
and evaluator) and learn about their role in the project. Nearly all the headteachers and Year 5 teachers 
from the 24 schools, along with other interested parties (Such as governors) attended this event. 


Also prior to randomisation in March 2014, headteachers of each school were briefed on the objectives 
of the programme. Consent was obtained from each school by means of a letter sent to the headteacher 
and the potential MathsFlip co-ordinator (the latter identified when the schools volunteered to 
participate). The letter explained the purpose and nature of the evaluation and was signed by the 
headteacher at each school once it was chosen to participate. 


Headteacher consent was obtained before randomisation. In January 2014 a letter was sent, via the 
schools, to parents and this included an opt-out form if a parent did not wish for their child to take part 
in the evaluation activities. Parent assemblies were also organized to provide further information on the 
programme and evaluation. (See Appendix 8 for the headteacher consent forms and Appendix 9 for the 
parent letter and opt-out forms.) 


Thirty laptops were delivered to each school. If schools were larger that single form entry schools the 
school either provided laptops to the children or classes shared the laptops that Shireland gave them. 
Control schools were loaned laptops, but only after the trial had finished. 
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Outcome measures 


KS1 mathematics assessments were used as the primary baseline measure. However, since KS1 SATs 
are assessed by the class teachers, an additional test—Progress in Maths (PiM)—was administered to 
establish baseline equivalence between treatment and control groups. 


There was some discussion about whether to use the PiM test (an online test developed by GL 
Assessment had been administered by teachers in March 2014 prior to randomisation under 
examination conditions) or use national Key Stage tests. It was not possible to obtain all the data from 
the pupils who sat the baseline test and so it was collectively decided—between the EEF and IEE—that 
the KS1 results would be the key baseline measure in the final intention-to-treat analysis and the PiM 
scores would not be used. 


The KS2 maths score was the primary outcome measure, obtained by the evaluator from the National 
Pupil Database. The schools sent the unique pupil numbers (UPNs) of the pupils in the study to the 
evaluator who requested the matched KS1 and KS2 scores for each pupil. These measures are high in 
contextual validity as they constitute the main indicators of pupils’ academic performance in England. 
Pupils in the intervention and control arms of the trial cohorts sat the KS2 tests in May and June 2015. 


The secondary outcomes were KS2 average point score, and English—defined as reading and writing 
scores at KS2. Writing and reading outcomes were chosen to examine the potential spill-over effect of 
MathsFlip to other subjects. The protocol also states that attendance data and pupil engagement with 
the flipped learning virtual learning environment would be collected. Shireland collected this data but it 
was not requested by the IEE at the time of analysis therefore it is not possible to include in this 
evaluation report. 


Statistical power was assessed using Optimal Design software. Based on prior experience with similar 
clustered analyses, assumptions and details were made as follows and included in the protocol: 


e number of schools: 24; 

e students per school per year group: 60; 

e classes per school: 2; 

e pre-post (KS1-KS2) correlation (Squared): 0.60; 
e intraclass correlation (KS2): 0.15; 

e criterion for statistical significance: p < 0.05; 


e Minimum Detectable Effect Size: in the protocol, 0.20 (calculations made after the study was 
finished suggest that the MDES at the protocol stage should have been 0.33 not 0.20 as stated 
in the protocol; the number of schools in the trial limits the power to detect an effect size of 
0.33); and 


e =power: 0.80. 
Randomisation 


As described above, the IEE selected 24 schools to participate. The lead evaluator matched schools in 
pairs based on similarities in prior attainment (percentage of pupils achieving level 4 or above in KS2 
maths scores in 2013), pupil premium eligibility, and proportion of FSM pupils (EVERFSM). The IEE 
data manager, who knew nothing about the specific schools, then assigned one school of each pair to 
the treatment group and the other to the control group using a random number generator. 
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Analysis 


The ITT (intention to treat) analysis of the primary outcome—pupils’ KS2 maths fine grade scores 
(KS2_KS2MATFG)—used multilevel modelling (MLM), an analysis in which pupils were nested within 
schools. Because randomisation was carried out using matched pairs, the pairings were accounted for 
by including them as an extra level in the analysis. This resulted in a three-level model, with Pupil nested 
within School nested within Pair as random effects. The MLM analysis used degrees of freedom 
associated with the number of schools, not the number of students. Pupils in schools randomly assigned 
to flipped learning were compared to those in the randomly assigned control group schools, controlling 
for their KS1 maths scores (KS1_MATPOINTS). 


Secondary analyses of pupils’ KS2 fine grade Average Point Score (KS2_KS2APSFG), fine grade 
reading (KS2_KS2READFG), and fine grade writing (KS2_KS2WRITTAFG) scores were conducted 
using the same model controlling for the corresponding subject’s KS1 points scores. The protocol also 
states that data on attendance would be included as a secondary outcome. However, this data was not 
requested by the evaluators so this analysis could not be done. 


Subgroup analyses were conducted for KS2 mathematics outcomes for boys, girls, and pupils eligible 
for FSM, controlling for their KS1 maths points scores. An analysis comparing high, average, and low 
achievers was conducted. The pupils were categorised as ‘high’, ‘average’, or ‘low’ achievers based on 
a division of the KS1 mathematics scores into three levels of similar size. 


It was also intended that an analysis of pupil engagement with the flipped learning VLE would take place 
to analyse if the degree of engagement had an impact on outcomes for those pupils in the treatment 
arm. We did not collect this data, therefore this analysis could not be conducted. 


The effect size was calculated using Hedges’ g. Since this was a cluster randomised trial, there are 
options for the calculation of the effect size (Hedges, 2007). Accordingly, we calculated the effect size 
using the difference between the arms in the adjusted means (accounting for pairing) divided by the 
unadjusted total standard deviation in the KS2 outcome. 


Implementation and process evaluation 


A process evaluation was conducted to develop an understanding of the context of the MathsFlip 
approach, of the day-to-day delivery of the intervention, and the teachers’ and pupils’ perceptions. Five 
research instruments were developed to provide a detailed picture of implementation of the programme 
and of perceptions of it. It also helped to inform and explain the impact evaluation findings. The five 
elements were: 


e interviews with implementation staff (see Appendix 1); 

e online teacher surveys (See Appendices 2 and 3); 

e telephone interviews with teachers (see Appendix 4); 

e school visits that included class observation (See Appendix 5); and 

e focus groups with teachers and pupils (see Appendices 6 and 7 respectively). 


Interviews with implementation staff 


In order to obtain a full understanding of the context and the rationale for MathsFlip, five in-depth face- 
to-face interviews were conducted with implementation staff from Shireland Collegiate Academy 
including the Headteacher, the Project Director, the Project Manager, the Director of Learning and the 
IT technicians who supported the project (See Appendix 1). 


The interviews were conducted to gain an in-depth understanding of the nature of the programme 
including its origins and development as well as the process for training and implementation. The 


interviews were conducted by two members of the evaluation team and were fully transcribed. The 
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transcripts were analysed via qualitative content analysis (Schreier, 2014). A coding frame was 
developed and the transcripts were analysed in a systematic way to develop a detailed understanding 
of: 


the design and development process; 

e the aims and intended outcomes of MathsFlip; 

e thecore and flexible components of the model; 

e how the approach can be differentiated from standard classroom practice; 
e training and implementation support offered to schools; and 

e barriers and potential challenges for implementation and delivery. 


Online staff surveys 


Two online surveys were administered to all Year 5 and Year 6 teachers in both treatment and control 
groups (those teachers who were implementing or were likely to implement MathsFlip in the future). The 
first survey was administered in April-May 2014. Results from this first survey provided a ‘baseline’ view 
of teachers’ perceptions regarding MathsFlip. The second survey was administered to the same 
teachers in May—June 2015, a bit later than planned to give more time for implementation of the 
intervention. 


Topics covered by the surveys included: 
e details of the school context and contexts for mathematics teaching; 
e teachers’ role, experience, age, and qualifications; 
e previous experience of other mathematics approaches or interventions; 
e current approaches to teaching mathematics; 
e views about methods for teaching and learning mathematics; 


e (for treatment schools) views about the programme, challenges, and successes in the MathsFlip 
approach; and 


e (for control schools) the extent to which new technologies are used in mathematics teaching 
and learning. 


The protocol also stated that ‘patterns of achievement in the schools generally and in mathematics as 
a subject’ would be collected. This was not collected via the process evaluation (but was via the impact 
evaluation). 


The survey findings were analysed longitudinally to compare views over one year, and cross-sectionally, 
prior to the start of the intervention and towards the end of the evaluation, in order to control for previous 
perceptions and to compare control and treatment schools. 


Telephone interviews 


Eleven telephone interviews were conducted in March 2015 with the flipped learning co-ordinator at 
intervention schools. The aim of these interviews was to collect detailed information about the 
implementation of the programme and challenges and successes up to the day of the interview (see 
Appendix 4). These were conducted later in the school year than outlined in the protocol to better capture 
what was happening in the implementation. Results from the interviews provided formative findings to 
the evaluation. Two members of the evaluation team conducted these semi-structured interviews 
following a predetermined set of questions based on topics featured in other research instruments, 
including the online surveys. When appropriate, the evaluators prompted the teachers to provide further 
details or additional examples. 
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School visits 


Four school visits to treatment schools were carried out by two members of the evaluation team to allow 
triangulation of the results. The visits included: 


e an observation of a mathematics lesson (30-60 minutes); 
e a30-minute focus group with teachers; and 
e a20-minute focus group with pupils. 


Four schools were randomly chosen and invited to participate by hosting a school visit. One of these 
schools declined a visit due to pressure from a forthcoming Ofsted inspection, and this was replaced 
with another randomly selected school. 


A pre-set observation sheet was developed using the cognitive dimensions of Bloom’s taxonomy (Bloom 
et al., 1956; see also Anderson and Krathwohl, 2001) as this is linked to the Higher Order Thinking Skills 
(HOTS) encouraged by MathsFlip and the principles of MathsFlip delivered in the training day. This 
observation sheet was used to evaluate the activities, engagement, and mathematical processes carried 
out during the lesson (See Appendix 5). The observation sheet was based on the aims of MathsFlip and 
was developed in consultation with Shireland’s project staff. 


Two schedules were developed to guide the discussions in the focus groups with teachers and pupils. 
The questions asked for views, challenges, and successes of the project as well as the extent to which 
the approach could be used in other subjects. The teachers were selected by the schools based on their 
availability (groups ranged from one to four teachers) and the students were randomly chosen from the 
observed lesson by the evaluator (groups ranged from four to ten pupils. 


Costs 


Costs were collected from the Shireland staff. These included costs for setting up the technology and 
the virtual environment resources, annual software licenses, as well as costs for training teachers and 
providing ongoing support. 


We considered the equipment costs for laptops and trolleys a prerequisite cost for two reasons. First, 
the laptops were loaned to the schools and at the end of the evaluation they returned them to Shireland 
so they could be used by the delayed-treatment control schools. Second, with the increasing use of 
laptops or tablets in schools, it is more likely that schools will already have this equipment. 


To calculate the cost per pupil, we divided the total cost over three years by the number of pupils at the 
time they were randomised. The total cost per pupil was divided by three to obtain a per pupil per year 
cost. 


School staff time was not estimated as an additional cost because the training was completed on 
scheduled days for professional development so supply cover was not required. The time for 
implementation of MathsFlip was during regularly scheduled teaching time. 


Timeline 


A two-armed randomised-controlled trial was conducted from spring 2014 to spring 2015, with 12 
schools in the treatment group and 12 in the control group. Schools participating in the treatment group 
received training and support in the MathsFlip intervention beginning in spring 2014. They implemented 
the MathsFlip approach from April 2014 through to April 2015, with pupils beginning in Year 5 and 
following on into Year 6. Thus the intervention overlapped two school (academic) years, but actually 
lasted one calendar year. Schools participating in the control group acted as a delayed treatment group 
and received the training and support for their Year 5 teachers from April 2015—one year after the first 
implementation cycle had begun. 
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Since the Key Stage results were our main outcome, we focus on these results in this report. A full 
timetable for the main project activities is presented in Table 1. 


Table 1: Timetable for the evaluation 


Date 


November—December 
2013 


January 2014 
March 2014 
April-May 2014 

April 2014 

April 2014—April 2015 
July 2014 

March 2015 

March 2015 
May—June 2015 

May 2015 

April 2015 


April 2014—April 2015 


Education Endowment Foundation 


Activity 


Recruitment of schools 


Teacher and parental consent sought 
Randomisation 

Online teacher surveys (baseline) 

Intervention with treatment schools began 
Intervention implemented in treatment schools 
Interviews with Shireland project staff 
Teacher telephone interviews 

School visits (observation and focus groups) 
Online teacher surveys (post-intervention) 

All treatment and control pupils sat KS2 SATs 
Delayed intervention with control schools began 


Intervention implemented in control schools 
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Impact evaluation 


Participants 


The recruitment process was very successful. Details of the school characteristics, presented in Table 
2, show that the control group had more pupils, on average, than the treatment group and a higher 
percentage of the control schools were academies (33% compared to 25%). A third of the control 
schools were urban schools, while none of the interventions schools were. The proportion of pupils 
receiving FSM was similar (23% for treatment schools versus 26% for control schools). A smaller 
proportion of pupils in treatment schools had English as an Additional Language (EAL) compared to 
control schools. The treatment schools had somewhat higher Ofsted ratings than the control schools. 
The control and treatment schools had similar KS1 average mathematics point scores. These averages, 
all under 16, were out of a maximum possible score of 27. 


A couple of characteristics might have benefitted the intervention group, those being the smaller size 
(average 380 pupils for intervention schools and 442 for control) and a smaller proportion of EAL pupils, 
and that none of the intervention schools were categorised as urban. However, the fact that the 
proportion of FSM pupils was quite close would make them seem more similar. 


Table 2: Baseline characteristics 
Variable Intervention group (oxelniige) Mel celel) Difference 


School-level n/N n/N 
‘(or 1=le (ol arer:|)) (missing) Percentage (missing) Percentage 


Ofsted rating: 

Outstanding 3/12 (0) 3/12 (0) 
Good 8/12 (0) 5/12 (0) 
Requires improvement 1/12 (0) 4/12 (0) 
Inadequate 0 0 


0 
Urban 0% 33% 
Town and fringe 100% 67% 


School-level 
(continuous) (missing) Mean n (missing) Mean 


KS1 maths score 12/12 (0) 15.75 12/12 (0) 15.44 
n/N n/N 


Pupil-level 
(or: (-Ye To) ater: l)) (missing) Percentage (missing) Percentage 


Eligible orFSM 871380 (0 saya? (0 
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Figure 1 presents the number of participants at each stage of the evaluation. At follow-up, data from 74 
pupils (25 from treatment, 49 from control) was unavailable. A sensitivity analysis was not conducted 
because the attrition rate was less than 6% and no school was eliminated from the main analyses. 
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Figure 1: Participant flow chart 


Recruitment 


Approached (school n = 52) 


Declined to participate 
(school n = 20) 


Assessed for eligibility 


(school n = 32) 


Excluded (school n = 8) 
not meeting inclusion 
criteria 
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Lost to follow- 


up 

(pupil n = 36 
iS missing KS1 
= and/or KS2 
= data) 
(o) 
LE 

Not analysed 
r) ee 
n 
> 
© 
= 
<x 


Allocated to intervention 
(school n = 12; 
pupil n = 578) 


KS1 and KS2 maths 
data 
(school n=12; 
pupil n= 542) 


Analysed 
(school n = 12; 
pupil n = 542) 


Allocated to control 
(school n = 12; 
pupil n = 636) 


KS1 and KS2 maths 
data 
(school n = 12; 
pupil n = 587) 


Analysed 
(school n = 12; 
pupil n = 587) 


Lost to follow- 
up 
(pupil n = 49 
missing KS1 or 
KS2 data) 


Not analysed 
(n=0) 


Table 3 demonstrates calculations of the minimum detectable effect size at different points in the 
evaluation: in the protocol, after randomisation, and in the ITT analyses. 
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Table 3: Minimum detectable effect size at different stages 


Protocol 


Randomisation 


Analysis 


N [schools/ 


pupils] (n = 
intervention; 
lam exe) ae) )) 


24 schools/ 
1,440 pupils 


(720; 720) 


Average 
cluster = 60 


24 schools/ 
1,214 pupils 


(578; 636) 


Average 
cluster = 51 


24 schools/ 
1,129 pupils 
(542; 587) 


Average 
cluster = 47 
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Correlation 
squared 
between pre- 
& post-test 
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Pair matching Power Alpha 


12 matched 
pairs of schools 


12 matched 
pairs 


12 matched 
pairs 


Minimum 
detectable 
effect size 

(12) S59) 
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Outcomes and analysis 


Overall analyses for primary and secondary outcomes 


The MathsFlip approach had little impact on the mathematics achievement of pupils at the end of KS2. 
It had more impact on the secondary outcomes, controlling for their KS1 scores (see Table 4), 
particularly on reading and writing. In each analysis the covariate was the corresponding KS1 score, 
with Maths KS1 being used in the analysis for KS2 Maths outcomes and the KS1 ReadWrite score being 
used in the analysis for both Reading and Writing outcomes and the Average Point Score. The KS2 fine 
grade score was used for all of the post-test outcomes. 


Missing Data 


For the primary analysis, 7% of the data is classed as missing. A sensitivity analysis was not conducted 
because the school-level attrition rate was low and no school was eliminated from the main analyses. 


Table 4: Analyses for primary and secondary outcomes 


Raw means Effect size 
Intervention group Control group 


n in model 
(excludes 
missing) 
(intervention; 
control) 


KS2 Maths 542 29.10 587 28.68 1,129 0.09 
(fine grade)| (36) (28.63; 29.56) (49) (28.26; 29.09) (542; 587) (-0.28; 0.46) 


n Mean (95% n Mean 
(missing) (o4)) (missing) (95% Cl) 


Hedges g 


Outcome | CEYZa 64 )) 


p-value 


KS2 
Average 545 28.75 587 28.28 1,132 0.21 
Point (33) | (28.36; 29.14) | (49) | (27.93: 28.64) | (542: 587) | (-0.20: 0.62) 
Score 
Reading (2) (28 eee 88) ae (27 ie 22) (942; 587 (-0 ne 57) 
(fine grade) ies ras oye .13; 0. 
wrting | G2 | ogc han | Bh | orzo | 48580 | po Pree 
(fine grade) ee a oa 


Subgroup analyses 


Further analyses were conducted for each sub-group: no significant impact of MathsFlip on KS2 maths 
scores was detected. Almost all outcomes had positive effect sizes but there was a fractionally negative 
effect of -0.01 for low achievers and -0.03 for high achievers. However, none of these effects were 
significant. (See Table 5 for the sub-group analyses.) 
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Table 5: Subgroup analyses for KS2 maths fine grade scores. 


Raw means Effect size 


Intervention group fofol alice) Mel geley e) 


nin model 
n Mean n Mean (intervention; Hedges g 
Subgroup (missing) (95% Cl) (missing) (95% Cl) control) (95% Cl) 


287 29.27 319 29.07 606 0.06 
(14) (28.60; 29.93) (11) (28.47; 29.66) (287; 319) (-0.32; 0.45) 


255 28.90 268 
(11) (28.26; 29.55) (13) 
195 28.76 231 
(8) (27.95; 29.57) (10) 


28.21 523 0.13 


(27.66; 28.77) | (255;268) | (-0.27:0.53) | 9-4 


27.46 426 0.10 
(26.87; 28.04) (195; 231) (-0.21; 0.41) 


Low 136 24.35 168 24.32 304 -0.01 es 
Achievers (2) (23.47; 25.23) (2) (23.61; 25.03) (136; 168) -0.30; 0.27 
28.41 183 28.12 337 Gas 
Achievers (27.87; 28.95) (0) (27.67; 28.57) (154; 183) 
High 32.24 236 32.28 488 -0.03 Gee 


Achievers (31.69; 32.79) (1) (31.76; 32.80) (252; 236) -0.30; 0.36 


Cost 


This section estimates the cost of a school purchasing MathsFlip. Table 6 outlines the estimated costs 
for schools delivering MathsFlip over three years. These costs include the costs of technology set-up 
and integration, virtual environment set-up, and Shireland staff time for training teachers, but assumes 
that the schools already have computers or tablets and basic internet services. 


The cost per pupil was calculated by dividing the total cost over three years provided by Shireland staff 
(£536,600) by the number of pupils at the time they were randomised (1,219), making £440.20. That 
estimated cost was divided by three to obtain a per pupil per year cost of £146.73. 
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Total cost per 


Type of cost years pupil over 3 
years 
Technology set 
up and integration Start-up cost 
and virtual £140,000 £140,000 £114.85 
environment set- 
up 
Annual licenses 
and ongoing Running cost per 
technological schoal £58,700 £176,100 £144.46 
support 
Training Start-up cost £4,500 £4,500 £3.69 
Ongoing . 
pedagogical ape nee sad £72,000 £216,000 £177.19 
support 
TOTAL £536,600 £440.20 


The equipment costs (£136,000 or £5,667 per school for 30 laptops and a trolley) were considered a 
prerequisite cost. School staff time was not provided by Shireland as there were no supply cover costs 
and the intervention took place during normal teaching hours. It should be stressed, however, that there 
are many costs, including teacher time, which are not included in these figures. It does not include the 
additional time that the intervention teachers spent planning lessons. They attended training, studied 
MathsFlip materials, and spent time assisting pupils with the new programme, though we do not have 
figure for these activities. 
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Process evaluation 


The results from the five research instruments that were administered provide a detailed picture of 
implementation of the programme and of teachers’ and pupils’ perceptions. These results come from 
the lesson observations, interviews with Shireland implementation staff, focus groups with teachers and 
pupils participating in the intervention, telephone interviews with teachers, and online surveys. Data 
collected from these instruments was analysed and combined by means of triangulation and is 
summarised below to provide information about how school staff and pupils viewed the MathsFlip 
approach. 


The teacher surveys, pre- and post-intervention, were one of the main sources of information for the 
process evaluation. Fifty-four teachers from 21 schools completed the baseline survey but only 11 
teachers from 8 schools completed the post-intervention surveys. As there were so few post- 
implementation surveys and not all of them completed by the same teachers as the baseline survey, no 
conclusions can be made about changes from baseline to post implementation. 


Implementation 


Two key elements were identified which indicated the likelihood of successful implementation: ongoing 
support for teachers and ample mathematics resources. 


Ongoing support for the duration of the intervention from Shireland staff—both technical and 
professional—was highly valued by teachers. Teachers reported that in the early stages of 
implementation they had concerns about their own lack of technical knowledge and that these concerns 
were relieved by the technical and professional support received. Technical problems with online 
connectivity and maintenance of the laptops were solved promptly by the IT support staff. The IT expert 
had the knowledge to sort out school-specific issues related to connectivity. The prompt responses 
translated into pupils having the adequate tools when needed and teachers focusing their efforts on the 
teaching aspects rather than sorting out technical problems. 


Professional pedagogical support was provided by the Shirelands project manager. After the initial 
training, she visited the schools on a regular basis to help teachers who requested it and also to provide 
training for new teachers joining the schools partway through the intervention. Teacher interviewees 
indicated that her constant support and use of coaching techniques was invaluable, especially at the 
beginning of the project when teachers were experiencing difficulties in implementing the programme. 


Mathematics resources provided in the MathsFlip platform were the second element identified by 
teachers as making the intervention successful. The maths support provided during the project included 
a suite of online resources linked to the programme of study. These resources included advice and 
guidance about how to use the MathsFlip methodology and links to individual online mathematics 
resources provided by the project partners such as MyMaths, Mangahigh, and MathsWatch. Teachers 
commented positively on the variety of resources and the ease with which they provided pupils with 
extra practice tailored to their level. They also commented on the immediate feedback received from 
the software on the areas where pupils had had problems. This allowed them to plan and prepare the 
in-class activities in advance that best suited pupils’ different abilities. The pupils, in their focus groups, 
also commented on how they enjoyed and benefitted from using these resources. 


Challenges 


There were two kinds of challenges faced during the implementation: technical and human. Regarding 
the former, teachers and pupils reported having problems with online access or with the laptops—both 
at school and at home. Teachers acknowledged their lack of IT knowledge to solve daily problems with 
the laptops. One even mentioned being ‘afraid of touching the laptops’. However, all teachers 
recognised that the support provided by Shireland had been crucial in the continuity of the intervention. 
They emphasised the quick response to their problems, sometimes within the same morning. At home, 
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the main challenge was internet access. Despite the technical problems, alternatives were found to 
provide pupils with the materials and activities if they could not access them at home. One common 
example across the schools was the use of after-school or lunch clubs. Students who did not have home 
internet access, or who had not done the work at home, were invited to participate in the clubs in order 
to ensure that all the pupils had covered the content necessary to participate in the maths lesson. One 
teacher also mentioned that some activities could be downloaded at school to avoid problems with the 
internet at home. 


The second group of challenges were the human ones. Within this group, the main concern of teachers 
was the lack of time to plan lessons—teachers reported that extra time was needed. However, they 
recognised, with the exception of one teacher, that the amount of resources and support provided by 
Shireland had been outstanding. 


Parents’ responses to the project were generally positive, with the exception of one school where 
teachers perceived a limited involvement from parents (this was not a reflection on MathsFlip—these 
parents generally lacked involvement with the school). Parents’ support is key to motivate and supervise 
students’ activities at home. This lack of involvement resulted in a small percentage of students 
(estimated at 10% to 15%) not doing the tasks at home. This proportion is less than the 25% of 
disengaged pupils reported by Loch and Borland (2014). 


Two further factors that played an important role in a successful implementation were mentioned in the 
teacher interviews. One teacher reported that it was difficult to ensure that all the staff (full time and part 
time) would have access to the same information during the implementation stage. Second, it was 
emphasised that pupil independence is necessary for the approach to work. Pupils are responsible of 
their own learning at home which is later transferred and applied at school. If pupils are not independent 
learners and fail to learn to organise their time, it can hinder the aim of the activities planned for the 
lessons. 


Overall, teachers found the intervention attractive and some mentioned that they are using similar 
approaches in other subjects such as history, geography, and science. They were confident in 
recommending the approach to other schools and some had even mentioned it to their local networks. 


Fidelity 


Because objective implementation fidelity data was not collected by the evaluators, we cannot 
definitively say how well the programme was implemented nor its effect on pupil achievement. This 
section reports on some indicators of implementation fidelity assessed through teachers’ attendance at 
the training sessions, direct observation of maths lessons in four of the treatment schools, and through 
Year 5 teachers’ views on the follow-up support provided by the Shireland project manager. A total of 
16 teachers from the 11 treatment schools attended the two-day sessions in spring 2014. Where new 
staff joined the project at later stages, the project manager delivered the same training sessions in the 
school. 


Two members of the evaluation team observed an hour-long maths lesson in each of four schools. 
These observations took place in March and April 2015. An observation form was used to reach 
consistency in the observed areas (see Appendix 5). The observation aimed to detect whether the use 
of Higher Order Thinking Skills (HOTS) were used in class as a result of the pre-learning at home via 
the MathsFlip platform. 


All the observed lessons included mixed ability groups. Pre-lesson tasks had been given to complete at 
home. The online tasks were mainly activities designed to review or test pre-learned concepts. It was 
evident from the observations that pupils had plenty of opportunities during the lesson to share what 
they had learned at home and to explain the strategies used to solve real-life problems. Enough time 
was also allowed for pupils to analyse the information of the problems posed in class. Deeper learning 
was observed when students solved the problems using both the analysis of information and their prior 
knowledge. 
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Learning activities catered for pupils with different abilities. It was observed that teachers provided tasks 
with different levels of difficulty so students could choose an activity suitable to their level and, if they 
succeeded, could challenge themselves and try a more difficult one. Teachers kept pupils engaged 
through the use of ‘challenged’ and ‘super-challenged’ activities. Pupils were also engaged through 
games and real-life activities, for example planning a trip to London. 


In-class activities were designed to create discussions of how to solve problems. Pupils had to 
consciously share with their peers the steps to solve a problem so they could demonstrate 
understanding of Knowledge and also help other pupils to address any misconceptions. This allowed 
them to break down complex problems into simpler parts. 


In summary, the data from the surveys, focus groups, and observations suggests that MathsFlip was 
delivered well and behaviour in the class reflected the key components of a flipped learning approach. 
This implementation overview does not suggest that failure to implement the intervention was 
responsible for the lack of impact on pupils’ achievement; however, there was limited data collected on 
actual implementation fidelity so we cannot be sure about this. 


Perceptions 


Teachers 


Teachers’ perceptions of the MathsFlip programme were very positive, as demonstrated in the surveys, 
interviews, and focus groups. Although most teachers said in surveys and interviews that the 
programme did not have a major impact on their teaching style, they recognised that the approach 
greatly influenced their planning. This change was perceived to be at least partly a consequence of the 
accurate pupil assessment gained before the pupils entered the classroom. Teachers accessed the 
online resources to see what their pupils did at home and how successful they were. Pre-assessment 
information helped teachers plan activities accordingly to support the weak areas faced by pupils in their 
homework. Moreover, teachers were able to plan differentiated activities suitable for each student 
resulting in a more personalised teaching experience. Teachers mentioned that they were more flexible 
with the working groups in the lessons. They could divide their class into smaller groups, based on the 
information received, to target smaller pupil groups and provide further assistance to the pupils that 
needed it the most. 


MathsFlip helped to cover and revise more areas than if the knowledge was exclusively delivered in the 
classroom. The pre-learning time at home provided teachers with extra lesson time to do more 
interactive activities and to help struggling students. Teachers mentioned they moved more quickly 
through tasks and that they could spend more time in application activities and problem-solving 
activities. They identified that they had a more investigative approach where they could design real-life 
maths problems for students to work on in class. It was also mentioned that the approach helped 
students to become more independent learners who were in charge of their own learning. This allowed 
students at different levels to recognise their weak areas and work on them. 


The changes in teaching practice reported by MathsFlip teachers in the post-intervention surveys were, 
however, also mentioned by control teachers: they, too, had confidence in their teaching and that 
lessons were learner-centred. The difference was that control teachers reported using ICT 39% of the 
time in maths lessons, while the MathsFlip teachers reported using it 50% of the time. Due to the limited 
data that this is based upon, we cannot be sure that this is representative of the whole sample. 


Pupils 

Both teachers and pupils reported on the surveys that pupils enjoyed working with laptops. In some 
schools the laptops were not only used at home but pupils had them available during the lessons to 
access and review concepts if needed. Some schools set up lunch-time and after-school sessions for 
pupils who either did not have internet access at home, or were too busy with other activities to do the 
work at home. 
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Teachers and pupils responded that there was an increase in engagement and enthusiasm due to the 
games-based approach of the programmes. Mangahigh, for instance, was one of the most popular tools 
for maths activities. Teachers mentioned that its competitive approach suited boys in particular. 


Overall, teachers and pupils who responded to the survey believed that MathsFlip generated individual 
learning even though the analyses of the outcome data did not reflect this. Teachers perceived that 
pupils’ participation increased in lessons. As a result of the extra practice, pupils were keener to 
participate in class. The extra practice and preparation at home, as well as the opportunity to review at 
their own pace, boosted the confidence of less able pupils and pupils from deprived backgrounds. Other 
teachers also perceived that high ability students benefitted from the programme because they were 
able to work on extension tasks while the teachers worked with less able students meaning that their 
learning was not slowed down. Teachers reported that pupils’ overall scores in practice tests had 
improved. As a side result, teachers felt students had gained ICT skills. 


Although the intervention was delivered to all the pupils at all levels of attainment, teachers considered 
that both less able and more able pupils benefitted more from the intervention. For the less able, it was 
felt that MathsFlip helped to boost their confidence. Teachers believed that because the students had 
to work at home they were more prepared during the lessons which resulted in more participation. It 
was also important that they could access the resources many times and work at their own pace. Also, 
pupils from deprived backgrounds were provided with an opportunity to have a laptop—access to 
technology which might otherwise have not been available to them. 


The more able pupils were felt to have benefited from the multiple resources available in the MathsFlip 
platform. Moreover, pupils of different abilities used the class site to communicate with each other and 
collaborate in order to solve their problems. In some schools, teachers reported that girls benefitted from 
the independent learning and self-pace work. A special mention was also made about the benefits that 
boys, kinaesthetic and visual learners, obtained from the games-based activities such as Mangahigh. 
The nature of the competitive activities appealed and helped to motivate these types of pupils which 
resulted in higher engagement. 


Formative findings 


Overall, the intervention was considered—by teachers, students, and parents—as being a success. 
However, two points were identified by participants that could help to improve the intervention. First, 
despite the constant support from the project manager, teachers felt that they would benefit from online 
support (Such as videos with model lessons) to review at home what had been learnt during the training 
sessions. Such support would also help teachers who are not available for training sessions or who like 
to work at their own pace. Second, planning and preparation time should be taken into consideration for 
the planning stages. Teachers reported that implementation and lesson preparation took a lot of their 
time; they therefore recommended implementing the programme in year-groups with lower academic 
workload, however, no objective data was collected on how much time this preparation took. 


Because of the small number of responses to the surveys (and often from different teachers) we cannot 
say whether there were changes over time. 


Control group activity 


From the survey data, it appears that the control teachers were very similar to the intervention teachers 
in their characteristics and teaching style. Even on the post-intervention survey, both groups of teachers 
reported that they employed a mixture of whole-class, group, and individual activities. They reported 
having a similarly high level of confidence in their teaching ability. There was no indication that the 
control teachers were implementing aspects of MathsFlip with their pupils. The only aspect in which 
they seemed to be different was that 50% of intervention teachers reported using ICT regularly in their 
classes compared to 39% of control teachers. However, this was only based on the 11 teachers who 


completed the post-intervention survey. 
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Conclusion 


| Key conclusions 


1. Children in MathsFlip schools made the equivalent of one additional month of progress in maths, 
on average, compared to children in comparison schools. The three padlock security rating 
means this result is moderately secure. 


2. The impact on maths was slightly higher for children eligible for Free School Meals than for all 
children in the trial. These results are less secure than the overall findings because of the smaller 
number of pupils. 


3. Children in MathsFlip schools made three additional months’ progress in reading and writing, on 
average, compared to children in the other schools. However, this result should be treated with 
caution. First, there is not an obvious route by which this maths intervention could improve 
literacy results so much more than maths results. Second, the relatively small number of schools 
involved increases the likelihood that we would see a result like this just by chance rather than 
due to the intervention itself. 


4. The majority of teachers in the trial were very positive about the flipped learning approach and 
the technical and professional support they received from Shireland staff. The process evaluation 
suggests this support was necessary for successful implementation. 


5. Some teachers experienced technical problems with the online platform which were generally 
dealt with quickly by the developers. Some pupils did not have internet access at home; this led 
some schools to set up homework clubs providing online access. 


Interpretation 


We can be moderately to highly confident that the impact seen was due to the intervention. However, it 
is interesting to note that there was more impact seen in elements of the curriculum not targeted by the 
intervention—writing and reading. One explanation might be that the intervention allowed teachers to 
spend more time preparing, planning, and assessing other subjects. However, without a more detailed 
process evaluation it is difficult to understand if this occurred. It should also be noted that the relatively 
small sample size means that the results from this trial will be more variable and could result in an 
overestimation of effects or a false positive (Type | error). 


MathsFlip received positive comments from the teachers and pupils that used it. Evidence from a small 
sample of lesson observations indicated that the approach was being implemented as planned. The 
intervention had a small impact on the mathematics achievement of pupils at the end of KS2. However, 
the estimated effect sizes were larger for reading and writing and for the average points score at the 
end of KS2, although these effects were not statistically significant. There were positive impacts of the 
intervention for some of the subgroups, including FSM pupils, on the KS2 mathematics scores. 


Access to MathsFlip was reported by some pupils to be an issue. This could represent a serious issue 
for the program if pupils were unable to access material at home prior to class. This did not seem to be 
an important issue because a number of schools scheduled lunch-time or after-school sessions during 
which pupils could access the online content and activities. However, we cannot be sure that these extra 
sessions sufficiently compensated for not having access to the programmes at home. 


To date, most studies evaluating flipped learning have been small scale, non-U.K. based, and without 
a comparison group. The findings from these studies suggest that a flipped learning approach to 
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teaching and learning is popular but does not provide improved attainment when compared to ‘business 
as usual’. 


It should also be noted that most of the research that has been conducted on flipped or blended learning 
has been in higher education classes, not primary schools as is the case for this evaluation. However, 
it seems that Shireland Collegiate Academy had experienced very positive results implementing the 
intervention in their secondary school, which they expected would generalise to upper primary school. 


Limitations 


This evaluation began with Year 5 pupils and followed these pupils into Year 6. Year 6 is a key year in 
U.K. education with pupils sitting high stakes national tests (KS2 SATS). As such, many schools put a 
lot of resources into their Year 6 classes, including high performing teachers, experienced teaching 
assistants, and extra support. It could be argued that testing MathsFlip against Year 6 classes with these 
extra resources was a hard test. However, one could also assume that schools in both conditions were 
also putting the same amount of resources into their classes. 


We had intended to collect objective data on school attendance and pupil engagement to determine an 
effect of dosage but this data were not collected. However, there was no indication of a difference 
between treatment and control school attendance. 


An insufficient amount of process evaluation data was collected and analysed to provide explanations 
as to why the intervention did not have the level of predicted impact on children’s mathematics 
achievement. 


Finally, the original power calculations from the protocol stated that the trial was powered to detect an 
effect of 0.20. However, when this was recalculated at the end of the study, an MDES of 0.33 was 
obtained. This is a more reasonable estimate with only 24 schools. This trial did not detect statistically 
significant effects on the primary outcome, but it should be noted that the MDES at the analysis stage 
was 0.37. 


Future research and publications 


Previous studies have suggested that flipped learning is effective, however these have been carried out 
in other countries (especially the U.S.A.) and in higher and secondary education contexts rather than in 
primary schools. Conducting a randomised controlled trial with this number of schools in one particular 
part of the U.K. was an important step forward. In addition, this evaluation focused on flipped learning 
in mathematics for Years 5 and 6 and numerous questions were raised by project participants about the 
applicability of this approach for other subjects and other age groups. 


Practitioner, as well as researcher, interest in this topic is on the increase. It would be useful to use the 
evaluation findings to inform future development of flipped learning. It would also be useful to hold 
seminars or conferences on the origins and definitions of flipped learning, with examples of effective 
implementation, so that teachers can be efficiently equipped to evaluate the approach in their 
classrooms. 
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Appendix 1: EEF cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. More information about the EEF’s approach to cost evaluation can be found here Cost 
ratings are awarded as follows: 


Cost rating Description 


ELELE Very low: less than £80 per pupil per year. 
ELEEEE Low: up to about £200 per pupil per year. 


Moderate: up to about £700 per pupil per year. 
High: up to £1,200 per pupil per year. 


ELEEE Very high: over £1,200 per pupil per year. 
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Appendix 2: Security classification of trial findings 


1. Criteria for rating: in each column highlight the relevant cell in green. 

2. Initial score: write how many padlocks the trial has received based on the first 3 columns (“x”) 
and highlight in green (initial score is the lowest rating from the first three columns — see 
guidance on security classification for more detail). 

3. Adjust: record adjustment for balance and threats for validity in the adjust column 

4. Final score: write the number of padlocks (“x @”) in the relevant cell and highlight in green 

Provide a brief summary of your classification, following the bullet point prompts below 


Criteria for rating Initial 
score 


Attrition* 


4a Fair and clear quasi- 
experimental design for 


comparison (e.g. RDD) with MDES < Adjustment 
appropriate analysis, or 0.3 11-20% for Balance 
experimental design with 

minor concerns about validity [ ] 


38 Well-matched comparison 
(using propensity score 
matching, or similar) or 


experimental design with 21-30% 
moderate concerns about 
validity 
Adjustment 
- Weakly matched comparison for threats 
or experimental design with MDES < 31-40% to internal 
major flaws 0.5 validity 
a [] ze! 
18a : ; 
Comparison group with poor 
or no matching (E.g. MDES < 41-50% 
volunteer versus others) 0.6 
a 
No comparator arr > | over 50% 


e Initial padlock score: RCT, attrition was 7%, but MDES was 0.37 at analysis = 3 padlocks 
e Reason for adjustment for balance (if made): no adjustment made 

e Reason for adjustment for threats to validity (if made): no adjustment made 

e Final padlock score: initial score adjusted for balance and internal validity = 3 


1 Attrition should be measured at the pupil level (even for clustered trials) and from the point of randomisation to 
the point of analysis. 
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Appendix 3: Interview Schedule for Shireland Collegiate 
Academy Staff 


1) How did the development of MathsFlip come about? 

a. What is the historical context for MathsFlip? 

b. Who was critical to the development of the programme? Was there an individual designer 
or a team? 

c. What influences did you draw on? Theories? Other programmes? Evidence? Own 
experience? 

d. How have the materials and training evolved? 

e. Have you undertaken any in-house evaluations of MathsFlip? 


2) What are the aims of MathsFlip? 
a. What outcomes is it designed to achieve? 
b. What is the problem that MathsFlip is trying to solve? 


3) Can you describe how MathsFlip works? 
a. What are the key components? 
b. How do those components achieve the desired aims of the approach? 
c. What is the role of technology in MathsFlip? 


4) How does MathsFlip differ from what traditionally happens in primary maths lessons? 
a. In what way does it ‘flip’ teaching and learning? 
b. How would you explain the ‘flip’ element to someone who has never heard of this term? 


5) How does the training and support package work? 
a. What training is provided to teachers? 
b. Who delivers the training? 
c. What support, if any, is provided on an ongoing basis to schools? 
d. Are there any measures in place to monitor implementation? 


6) What are the main challenges in implementing MathsFlip and what have Shireland done to 
overcome those challenges? 


7) What do you feel are/will be the main benefits of MathsFlip for participating schools and pupils? 


8) How would you assess ‘success’ for the MathsFlip programme? What are you looking to achieve? 
What is the long-term vision for MathsFlip? 
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Appendix 4: Online Teacher Survey (pre-intervention) 


Introduction 


This short survey is for teachers taking part in the University of York’s evaluation of the Flipped 
Learning programme (also known as MathsFlip). It should take no longer than 15 minutes to complete 
and your answers will be kept confidential to the research team. Please also note that individual 
teachers and/or schools will not be identifiable when the results of this survey are reported. If you 
experience any difficulties completing this survey please contact Sarah Blower on 01904 328107 or 
Peter Rudd on 01904 328163. 


Some information about you 
Q1. Your name 
Free text 


Q2. Your school’s name 
Free text 


Q3. Which of the following most accurately describe your role in the school? (Tick all that apply) 
Year 5 teacher 

Other Year group teacher 

Maths co-ordinator/subject leader? 

Other (write in) 


Q4. Your gender: 
Female 
Male 


Q5. Your age 
Under 25 years 
25-34 

35-44 

45-54 

55 and over 


Q6. How long have you been teaching? 
Less than a year 

1-5 years 

6-10 

11-15 

15+ 


Q7. What is your highest qualification in Mathematics? 
Up to GCSE or equivalent 

A level 

Degree 

Masters/PhD 

Other (write in) 


Q8. On average, how many hours of Mathematics do you teach each week? 
Less than an hour 
1 hour 
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2 hours 
3 hours 
4 hours 
5 hours 
6 or more hours 


Your school and the context for mathematics teaching 
Q9. Does your school have specialist mathematics teachers? 


Yes 
No 


Q10. Do you plan mathematics lessons collaboratively with other teachers? 
Yes 
No 


Q11. Do teaching assistants help in mathematics lessons? 
Yes 
No 


Q12. If yes, do any of the teaching assistants have mathematics qualifications? 
Up to GCSE or equivalent 

A level 

Degree 

Masters/PhD 

Other (write in) 


Q13. Approximately what proportion of your mathematics lessons involve the direct use of ICT (e.g. 
computers, laptops, tablets, and interactive whiteboards)? 


Slider showing percentage range 


Q14. How often do you use the following resources in mathematics lessons? 


All lessons | Most Some A few Never 
lessons lessons lessons 


Maths textbooks 


Maths worksheets 


Maths websites 


Videos 


Interactive whiteboard 


Virtual learning environments (VLE) 


Handheld tablets or devices 
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Smart phones 


Laptops or PCs 


Your views about methods for teaching and learning mathematics 
Q15. In your view, how useful are the following teaching resources? 


Very Mostly Somewhat | Not Don't 
useful useful useful useful know 


Maths textbooks 


Maths worksheets 


Maths websites 


Videos 


Interactive whiteboard 


Virtual learning environments (VLE) 


Handheld tablets or devices 


Smart phones 


Laptops or PCs 


Q16. What do you feel about the availability of resources for primary mathematics teachers? 
Too many resources 

Plenty of resources 

Sufficient resources 

Not enough resources 


Q17. On ascale of 1 to 10, how confident do you feel about teaching mathematics? 
1 No confidence 
2 


NO 1 B® W 
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8 
9 
10 Extremely confident 


Q18. How would you describe your teaching methods (e.g. mainly whole class with some small 
groups, or a mixture of whole class, group and individual activities)? 
Free text 


Q19. On ascale of 1 to 10, how learner-centred would you say your teaching is? 
1 Very teacher-centred 


10 Very learner-centred 


Q20. Had you heard of Flipped Learning before joining this project? 
Yes 

No 

Not sure 


Q21. Have you implemented Flipped Learning before? 
Yes 

No 

Not sure 


Q22. Why were you initially attracted to Flipped Learning? 
Free text 


Q23. How do you think Flipped Learning might benefit your students? 
Free text 


Q24. Do you anticipate any challenges in implementing the Flipped Learning approach? 
Yes, because...... 


No, because........ 


Thank you for taking the time to complete this survey. 
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Appendix 5: Online Teacher Survey (Post-intervention) 


Q1 Maths Flip This short survey is for teachers taking part in the University of York’s evaluation of 
the Flipped Learning programme (also known as MathsFlip). It should take no longer than 15 minutes 
to complete and your answers will be kept confidential to the research team. Please also note that 
individual teachers and/or schools will not be identifiable when the results of this survey are reported. 
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If you experience any difficulties completing this survey please contact Berenice Villanueva on 01904 
328108. 


Q2 Some information about you 
Q3 Please tell us your name 
Q4 Your school's name 

Q5 Your gender 


O Male 
O Female 


Q6 Your age 


Under 25 years-old 
25 - 34 

35 - 44 

45 - 54 

55 and over 


OQOQOOO 


O 


7 How long have you been teaching? 


Less than a year 
1-5 years 
6-10 

11-15 

15 or more 


OQOOO0O 


O 


8 What is your highest qualification in Mathematics? 


Up to GCSE or equivalent 
A level 

Degree 

Masters/PhD 

Other (Please specify) 


9 On average, how many hours of Mathematics do you teach each week? 


OQOO0O0 


O 


Less than an hour 
1 hour 

2 hours 

3 hours 

4 hours 

5 hours 

6 or more hours 


OCOOQOO000 


Q10 Some information about your Mathematics teaching 
Q11 Ona scale from 1-10, how learner-centred would you say your teaching is? 


very teacher-centred 1 


QOWOQOQOQO0000 
CHONDAONRWN 


very learner-centred 10 


Q12 How, if at all, has your Mathematics teaching changed from the period before you had any Maths 
Flip training to the present? (TREATMENT) 
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Strongly Not sure 


agree 


| talk at the 
class less 
than I used to 


My lessons 
include a 
bigger range 
of activities 
than before 


| use 
ICT/computer 
s more 
effectively 
(for teaching 
and learning) 
than | used to 


Pupils now 
take a more 
active part in 
lessons than 
they did 
previously 


Pupils work 


together more 
often than 
they used to 
do 


My lessons 
are more 
interesting 
than they 
used to be 


| feel better 
motivated to 
teach than | 
did before 


Pupils ask 
more 
questions 
than they did 
previously 


| provide 
more 
opportunities 
to use Higher 
Order 
Thinking 
Skills (HOTS) 
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Strongly 
Disagree 
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I tailor my 
class for 
different 
abilities by 
including 
differentiation 
activities 


Pupils do 
more working 
out of things 
for 
themselves 
than they did 
before 


All round, | 
am amore 
effective 
teacher than 
before 


All lessons Most 
lessons 


Transfer of 
knowledge (i.e. 


from teacher to 
pupils) 


Discussions to 
clarify 
knowledge 
acquired 
outside the 
classroom 


Discussions to 
address 
misunderstandi 
ngs of concepts 
learnt before 
the lesson 


Activities to 
demonstrate 
understanding 
of knowledge 


Experiments 


Analysis of 
problems 


Break down of 


ideas into 
simpler parts 
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A few 
lessons 
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Comparison 
and contrasts of 
concepts 


Summary of 
ideas or new 


concepts 


Combination of 
knowledge that 
leads to new 
knowledge 


Draw 
conclusions 
from the lesson 


Q14 To what extent do you feel Maths Flip has changed the way you teach Mathematics? Please rate 
this on the following scale, where 1 means ‘no change at all’ and 10 means ‘Maths Flip’ has 
completely transformed the way | teach' (TREATMENT) 


Q15 Please indicate the extent to which you agree with each of the following statements. The 
experience of the Maths Flip programme has... (TREATMENT) 


Strongly Agree Not sure Disagree Strongly 
Xe} c-1-) DEY: Xe] c-1-) 


... improved 
my 
confidence as 
a teacher 


... brought 
about more 
variety in my 
teaching 


styles 


... helped me 
to plan 
lessons 
better 


..ledtoa 
better 
exchange of 
ideas in my 
classroom 
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... Improved 
my 
classroom 
management 
skills 


... enabled 
me to 
facilitate 
more 
collaborative 
learning 


. has 
ensured that | 
make more 
effective use 


Q16 Some information about your pupils 


Q17 To what extent do you feel your pupils have benefited from Maths Flip? Please rate this on the 
following scale, where 1 means ‘no benefit at all’ and 10 means ‘Maths Flip has greatly improved their 
learning experience’: (TREATMENT) 


Q18 How confident are you that your pupils’ results in maths will improve as a result of Maths Flip? 
Please rate this on the following scale, where 1 means ‘It won't have any effect’ and 10 means ‘Their 
results will be positively improved’ (TREATMENT) 


Q19 Please tell us about one Mathematics lesson that you have delivered where you had perceived 
your pupils to be highly engaged 


Q20 a) What was the topic of the lesson? 

Q21b) What activities did you use in the lesson? 
Q22c) Why do you feel the pupils were engaged? 
Q23 Some information about Maths Flip (TREATMENT) 


Q24 Ona scale from 1 to 10, where 1 means ‘I won’t use them at all’ and 10 means ‘I will always 
include them in my lesson’, how likely is it that you will keep using the skills learnt while using the 
Maths Flip programme? (TREATMENT) 


Q25 On a scale of 1 to 10, where 1 means ‘not at all useful’ and 10 means ‘extremely useful’, how 
would you rate the Maths Flipped programme? (TREATMENT) 


Q26 How likely are you to recommend the use of Maths Flip to other schools? (TREATMENT) 


Q27 Is there any further comments you would like to make about your experience of Maths Flip? (If 
so, please write your comments in the space below) (TREATMENT) 


Q28 Which of the following most accurately describe your role in the school? (Tick all that apply) 
UL) Year 5 teacher 

Year 6 teacher 

Other year group teacher 

Maths co-ordinator/subject leader 

Other (Please specify) 


Q) 
Q) 
Q) 
Q) 


Q29 How, if at all, has your Mathematics teaching changed from last year to the present? 
(CONTROL) 
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Strongly Not sure 


agree 


Agree 


| talk at the 
class less 
than I used to 


My lessons 
include a 
bigger range 
of activities 
than before 


| use 
ICT/computer 
s more 
effectively 
(for teaching 
and learning) 
than | used to 


Pupils now 
take a more 
active part in 
lessons than 
they did 
previously 


Pupils work 


together more 
often than 
they used to 
do 


My lessons 
are more 
interesting 
than they 
used to be 


| feel better 
motivated to 
teach than | 
did before 


Pupils ask 
more 
questions 
than they did 
previously 


| provide 
more 
opportunities 
to use Higher 
Order 
Thinking 
Skills (HOTS) 
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Strongly 
Disagree 


Al 
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I tailor my 
class for 
different 
abilities by 
including 
differentiation 
activities 


Pupils do 
more working 


out of things 
for 
themselves 
than they did 
before 


All round, | 
am amore 
effective 
teacher than 
before 


Q30 Approximately what proportion of your mathematics lessons involve the direct use of ICT (e.g. 
computers, laptops, tablets, and interactive whiteboards)? 


Time 
Q31 How often do you use the following resources in Mathematics lessons? 


All lessons Mostlessons Some A few Never 
lessons Ketotoxe) aks} 


Math 
textbooks 
Math 
worksheets 
Math 
websites 


Interactive 
whiteboard 


Virtual 
learning 
environments 


Handheld 
tablets or 
devices 
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Laptops or 
PCs 


Q32 How would you describe your teaching methods (e.g. mainly whole class with some small 
groups; or a mixture of whole class, group and individual activities?) 


Q33 On a scale of 1 to 10 how confident do you feel about teaching Mathematics? 
Q34 In your view, how useful are the following teaching resources? sina 


Very useful Mostly useful Somewhat Not useful Don't know 
useful 


Maths 
textbooks 
Maths 
worksheets 
Maths 
websites 


Videos | 


Interactive 
whiteboard 


Virtual 
learning 
environments 


Handheld 
tablets or 
devices 


Smart Smartphones) 


Laptops or 
PCs 
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Appendix 6: Teacher Telephone Interview Schedule 


10. 


11. 


12. 


Do you feel you have had the necessary support and training to deliver the Maths Flip programme 
to your pupils? 


How, if at all, has your teaching changed from the period before you used Maths Flip? What do 
you do differently now? 


Are you able to give me any clear examples of how a Maths Flip activity has engaged pupils or 
improved their understanding? 


Have you seen any changes in your pupils’ attitudes since they started using Maths Flip? 


Are there any groups or types of pupils who you think have benefited more from the programme? 
(e.g. high achievers, low achievers) 


Do you think pupils’ results in maths will improve as a result of Maths Flip? (If yes, why do you 
say that? Do you have any evidence for that?) 


How do you perceive parents’ response to the programme? 


What do you, as a teacher, think about the programme? Could you tell me if and how you have 
benefited from the programme? 


What have you, as a teacher, gained by flipping your class? 


What have been the main problems you have faced while participating in the Maths Flip 
programme? 


Do you think the use of Flipped Learning could be successfully translated to other subjects? 


Would you recommend the use of Maths Flips to other primary schools? Why? 
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Appendix 7: Lesson Observation Sheet 


School name Time of observation 
Teacher's name Class / Year 

Topic 

Observer’s name Number of pupils 


Resources in class: 


(i.e. laptops, whiteboards, handouts, etc) 


Any comments on pupil characteristics: 


(e.g. mixed ability, any SEN children, gender spread) 


Lesson objectives: 


a 


Details of the nature of the Flipped Learning 


(i.e. what preparation did pupils do before the lesson?) 


Overview of main lesson activities 


Please describe any examples of activities where you observed the use of HOTS during the lesson 


Were there any problems or issues during the lesson? Please provide brief details 


Please describe / list any examples of activities that promoted pupils’ engagement 


|. Aims of Flipped Learning - Please tick as many options as you observe during the lesson. 
1st box — yes, definitely; 2nd box — yes, to some extent; 3rd box — no, limited 
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The activities in the lesson... 


provide opportunities to deepen the learning process 


allow the use of higher order thinking skills (HOTS) 


allow students to share the knowledge learned before the lesson 


provide plenty of time to refine HOTS 


include differentiation activities 


Il. | Bloom’s taxonomy - Which of the following processes can be observed during the lesson? 


Knowledge 
transfer of knowledge (From teacher to pupils) 


Understanding 
discussions to clarify the acquisition of previous knowledge 


discussions to address misconceptions of concepts learnt before the lesson 


Application 


activities to demonstrate understanding of knowledge 


experiments 


Analysis 


analysis of problems 


break down of ideas into simpler parts 


comparisons and contrasts of concepts 


Synthesis 


summary of ideas/new concepts 


combination of knowledge that leads to new knowledge 


Evaluation 


conclusions 


discussion of arguments 


evaluation of learnt concepts 
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Appendix 8: Schedule for Focus Groups with Teachers 


Good morning. Thank you very much for your time. My name is .... | come from the IEE at the 
University of York. | am here because we are interested in knowing your views about the Maths Flip 
project. Your comments are relevant to the independent evaluation of the project. [Ask for permission 
to record the discussion]. All your comments today will be anonymised and only the researchers at 
IEE will have access to your comments. 


To be completed before the interview 


School: Date of the visit: 


No of teachers: Visitor’s name: 


Role (select and write the number of people) 


Year 5 teacher 


Views on the programme 
1. How, if at all, has your teaching changed from the period before you used Maths Flip? What 


do you do differently now? 
2. Have the pupils’ attitudes changed in the classroom? How? 
3. How do you perceive your pupils’ engagement in class? Has it changed? How? 
4. How do you perceive pupils’ engagement at home? 
5. How do you perceive parents’ response to the programme? 


Weaknesses 
6. What do you consider have been the main challenges of the programme? Prompt with: 
problems with pupils not doing the tasks at home? 
any technical problems? 
identified any gaps in the programme or need for extra training? 


Strengths 
7. Could you tell me how you, as a teacher, have benefited from the programme? What have 


you gained by flipping your class? 

8. Are you able to give me any clear examples of how a Maths Flip activity has engaged pupils 
or improved their understanding? 

9. Are there any groups or types of pupils who you think have benefited more from the 
programme? (e.g. high achievers, low achievers) 

10. Do you think pupils’ results in maths will improve as a result of Maths Flip? (If yes, why do you 
say that? Do you have any evidence for that?) 

11. In summary, what do you think of Maths Flip overall? Would you recommend the use of 
Maths Flips to other primary schools? Why? 

12. Do you think the use of Flipped Learning could be successfully translated to other subjects? 


Thank you very much for your participation and your time. Your answers are really valuable to the 
evaluation of the programme. 


47 
Education Endowment Foundation 


Flipped Learning 


Appendix 9: Schedule for Focus Groups with Pupils 


Hello, I’m from the University of York and we are carrying out a project about maths in primary 
schools. Your teachers have agreed that we can come and talk to you about your maths lessons. | 
would be very interested in knowing a bit more about the work you are doing in mathematics. 


[| hope you don’t mind if a record this discussion, just so that | don’t have to write it all down]. 


1. What do you think of working with laptops at home/school? 


2. What are the tasks/activities you have to do at home with the laptops? Could you give me 
some examples? 


3. Do you like this kind of homework? What do you like about it? 
4. Could you name one of your favourite activities at home? 


5. What do you do if you do not understand something from the tasks that you have to do at 
home? 


6. Could you name one of your favourite activities in the maths lesson? 

7. Which maths activities did you not like this year? Why? 

8. Have you had any problems with the laptops? Which ones? How did you solve it? 
9. Does you teacher give you enough help if you have not understood something? 


10. Which maths lessons do you prefer, the ones you have had this year or the ones in your 
previous year? Why? 
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Appendix 10: Information and consent forms for 
headteachers and parents 


INFORMATION SHEET / OPT OUT FORM FOR HEADTEACHERS 
INSTITUTE FOR EFFECTIVE EDUCATION, THE UNIVERSITY OF YORK 


Flipped Learning Evaluation 
Letter to Headteachers 
Dear Headteacher 


As you may be aware Shireland Collegiate Academy is planning to implement a new project called 
Flipped Learning with around 20 primary schools in the Birmingham and Black Country area. 
Shireland Collegiate Academy has much experience in this area and the project has great potential to 
improve the engagement and attainment of children in Mathematics and in other areas of the 
curriculum. 


The Flipped Learning project is being funded by the Education Endowment Foundation (EEF) and is 
being independently evaluated by a team from the Institute for Effective Education at the University of 
York. This letter is a request for you to and your school to take part in this project and the associated 
evaluation. 


In January 2014, when the project will commence, the evaluation team will work with Shireland 
Collegiate Academy to start collecting information about the impact of Flipped Learning, which will be 
delivered in Year 5 Mathematics. 


Researchers at the Institute for Effective Education will conduct a trial of the approach to assess 
whether Flipped Learning improves children’s mathematic attainment. They will randomly assign 
participating schools either to the intervention (experimental group) or to a group of schools which will 
continue with their regular mathematics teaching (control group) for 12 months. Control schools will 
be offered the Flipped Learning intervention after the first 12-month phase, so they will not miss out 
on the project. 


The requirements of you and your school would be as follows: 


School groupings. 24 participating primary schools will be randomly divided into 12 
experimental schools (using Flipped Learning) and 12 control schools (not using Flipped 
Learning, but doing Year 5 Mathematics as normal) and we need you to agree to be in either 
group. There is a 50-50 chance you will be in the experimental group. Randomisation is an 
important part of the research and it helps us to obtain robust, relaiable and valid findings. If 
your school is in the control group, your Year 5 will not receive the Flipped Learning 
intervention this year, but it will be offered to you next year. 


Pupil tests. At the start and end of the intervention pupils in both the experimental and 
comparison classes will complete a standardised online mathematics test. The first test (the 
pre-test) will be taken by Year 5 pupils in January and the post-test will be taken by the same 
pupils (who will have progressed into Year 6) around December 2014. Online tests (GL 
Assessment’s Progress in Mathematics) will be used and Shireland staff will assist with their 
administration, so there should be no extra burden on your staff. Marking is carried out 
automatically in this system and your school will receive feedback on the test results. 


Staff survey. The evaluation will also involve a short online survey of teachers/school staff. 
We will ask the two members of your staff most closely involved in the Flipped Learning 
project to complete a questionnaire (taking around 15-20 minutes) in order to obtain their 
perspectives on the project. 


Researcher visits. In addition we will make a request to visit four participating schools in 
order to observe Flipped Learning in action and to talk to pupils and staff about its 
effectiveness and impact. We would appreciate your willingness to host a visit if sucha 
request is made. 
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We should like to stress that your participation in the evaluation is entirely voluntary and you can 
withdraw at any time. Confidentiality and anonymity are assured — no school or individual will be 
identified in any report on this evaluation. The pupil test data will be shared with the EEF in an 
anonymised format. The evaluation proposal has been approved by the University of York’s Ethics 
Committee. 


We do hope you will be able to assist us with this important project. The study will provide valuable 
information about the effectiveness of this innovation and about its potential for improving 
mathematics teaching more generally. 


Please complete and return the form below if you agree to take part in the Flipped Learning study and 
evaluation. 


Sir Mark Grundy Dr Peter Rudd 
Executive Principal Principal Investigator 
Shireland Collegiate Academy Institute for Effective Education 
Berrick Saul Building 
University of York 
Heslington 
YO10 5DD 


Please complete and return the form below if you agree to take part in the Flipped Learning 
evaluation. We would also appreciate the name of a Year 5 Teacher who could act as a key 
contact person for the evaluation team. 


| agree for my school to take part in the Flipped Learning 
study. 


Name of Head Teacher: 


Signature of Head Teacher: 


| agree to take part in the Flipped Learning study. 


Name of Year 5 Teacher: 


Signature of Year 5 Teacher: 


Year 5 Teacher email address: 


This completed form can either be sent electronically to peter.rudd@york.ac.uk, or by post to Peter 
Rudd at the address given above. 
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Appendix 11: Information sheet/opt-out form for parents 


INSTITUTE FOR EFFECTIVE EDUCATION, THE UNIVERSITY OF YORK 
Flipped Learning Evaluation 
Information Sheet for Parents/Guardians 
Dear Parent/Guardian, 


We would like to request your permission for your son/daughter to take part in an educational 
evaluation study. The following information explains why the research is being done and what it would 
involve. 


What is the Institute for Effective Education? 
The Institute for Effective Education (IEE) is part of the University of York. It aims to find out what 
works in teaching and learning and why, and then use the evidence to improve education. 


What is the purpose of this study? 

This study is being done to see if using a Flipped Learning approach in Mathematics helps pupils to 
achieve better results. The Flipped Learning project involves the use of tablets and is being delivered 
by Shireland Collegiate Academy with a number of primary schools in your area. 


Why is my son/daughter’s class participating? 
We will conduct this study with current Year 5 classes in 24 primary schools in the Black Country. The 
head teacher of your son/daughter’s school has agreed to participate in this study. 


What will my son/daughter’s participation be? 

In January 2014, and again later in the year, the current Year 7 students will do a standard online 
Mathematics test. The tests are widely used by schools. The tests will be administered by Shirelands 
Collegiate Academy staff working closely with your child’s class teacher. We will use the test results to 
see if the Flipped Learning Programme has improved pupil learning, and the results will also be fed 
back to your child’s school. 


Does my son/daughter have to take part? 

You may choose not to permit your son/daughter’s test scores to be used in the evaluation. If so, 
please complete and sign the attached opt-out form by Friday 10 January. The right to withdraw your 
son/daughter from the testing will be fully respected. 


What are the disadvantages and risks of taking part? 

There are no known disadvantages or risks in participating in this study. The test data will be securely 
stored and managed, scored electronically. Pupils names’ will be removed so that the scores are 
anonymised and the data will be shared with the Education Endowment Foundation (who are funding 
the Flipped Learning project). Teachers will continue to teach to the usual lesson objectives 
throughout the evaluation period. 


What are the possible benefits of taking part? 

By participating in this study, your son/daughter will experience an up-to-date online approach to 
learning Mathematics. The information gained from this study may influence how your son/daughter 
and others will be taught more effectively in the future. 


What happens when the research stops? 
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IEE researchers will analyse the test scores to determine the overall effectiveness of the Flipped 
Learning Programme. Scores for individual pupils and classes will be kept confidential. When the 
research is over, the school will receive a report that will show if Flipped Learning makes a difference 
to pupils’ achievement. 


Will my son/daughter’s information be kept confidential? 
Yes. Student’s names will be replaced with code numbers. No individual student’s data will appear in 
any report. The research team will not have access to student or parent names. 


What if there is a problem? 
If you have a concern or question about your son/daughter’s participation in this study, please contact 
Peter Rudd (e-mail: peter.rudd@york.ac.uk) Tel: 01904 328163. 


Yours Sincerely, 


Dr Peter Rudd 

Reader in Education 

Institute for Effective Education 
University of York 
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THE UNIVERSITY Of fork 


INSTITUTE FOR EFFECTIVE EDUCATION 


Parent/Guardian opt-out form 


If you do not permit your son/daughter’s test scores to be used in the study, please complete this 
form and return it to your son/daughter’s form teacher or the school reception by by Friday 10 
January 2014. 


| do not wish my son/daughter’s test scores be used in the study. 


Student's: name?) ted iss ei te a ae ee 
(Please print clearly). 


Form: Teacher's: NAMO@? secs ak ceed oot ols caused osaae co sacaed saat onsviedbsotacateabasouadeennus ddvbed saddens sb uceiapeeaok 


SCHOO)! ideicieacanien Fes ceva wi bens Pica ava tides dhatiaciddnavacaiave Ueiweaarwaalevaswideuacdeatavactedeecwndaeans 


Parent’s/Guardian’s NAME: ............ cee cence eee eetee ee eee tne e eter ea aeeeeeeteaeeeesneneaeeeeetnnaaeeeeennaaeeees 
(Please print clearly) 


Parent’s/Guardian’s Signature: ............. cece etter eee teeter eenae eee eeeaaeeeeeeeaaeeeesenaeeeeenenaaes 


Daten iis iisieieied As wedentiuase aoe dnaaaa eas piddeedaadne Fan bois Maaatne (anshtebaaueian nae aaa eases 
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You may re-use this document/publication (not including logos) free of charge in any format or medium, 
under the terms of the Open Government Licence v3.0. 


OGL This information is licensed under the Open Government Licence v3.0. To view this licence, 
visit http:/www.nationalarchives.gov.uk/doc/open-government-licencel 


Where we have identified any third party copyright information you will need to obtain permission from 
the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 


A Education 


Endowment 
Foundation 


The Education Endowment Foundation 

9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 
www.educationendowmentfoundation.org.uk 


